{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Caffe2 python sub-imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Python wrapper modules**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" cellpadding=\"3\" cellspacing=\"0\"  style=\"border:black; border-collapse:collapse;\"><tr><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Core</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Brew</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Model&nbsp;Helpers</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>CNN&nbsp;Helpers</b></td></tr><tr><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Caffe&nbsp;Translator</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Operators</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Memory&nbsp;Manager</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Distributed&nbsp;Computation</b></td></tr><tr><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Model&nbsp;Downloaders</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Net&nbsp;Operations</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Base&nbsp;Optimizers</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Base&nbsp;Layers</b></td></tr><tr><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Base&nbsp;Regularizers</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Workspace</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Predictor</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Examples</b></td></tr><tr><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Base&nbsp;LSTM&nbsp;and&nbsp;RNN&nbsp;Cells</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Data&nbsp;I/O</b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b></b></td><td  style=\"border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b></b></td></tr></table>"
      ],
      "text/plain": [
       "<ipy_table.ipy_table.IpyTable at 0x7f684409bd90>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from matplotlib import pyplot\n",
    "import numpy as np\n",
    "import os\n",
    "from ipy_table import *\n",
    "from IPython.display import Markdown\n",
    "\n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "\n",
    "printmd(\"**Python wrapper modules**\")\n",
    "solvers = [\n",
    "    ['Core', 'Brew', 'Model Helpers', 'CNN Helpers'],\n",
    "    ['Caffe Translator', 'Operators', 'Memory Manager', 'Distributed Computation'],\n",
    "    ['Model Downloaders', 'Net Operations', 'Base Optimizers', 'Base Layers'],\n",
    "    ['Base Regularizers', 'Workspace', 'Predictor', 'Examples'],\n",
    "    ['Base LSTM and RNN Cells', 'Data I/O', '', '']\n",
    "    \n",
    "\n",
    "];\n",
    "make_table(solvers)\n",
    "set_row_style(0, bold=True)\n",
    "set_row_style(1, bold=True)\n",
    "set_row_style(2, bold=True)\n",
    "set_row_style(3, bold=True)\n",
    "set_row_style(4, bold=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Let's discuss each module in detail**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(\"**Let's discuss each module in detail**\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Core\n",
    "\n",
    "As the name suggests, most of the core parts in caffe2-python are instantiated using this module. The parts where core plays a few important roles, are, \n",
    "\n",
    "1) Creating operators \n",
    "\n",
    "2) Initializing and Managing networks\n",
    "\n",
    "3) Referencing data-blobs and assigning namescopes for proper management\n",
    "\n",
    "4) Setting device options (as seen in previous video)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.\n",
      "WARNING:root:Debug message: No module named caffe2_pybind11_state_gpu\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Core Imported**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parent Functions of core are: \n",
      "\n",
      "['BlobReference', 'C', 'CreateOperator', 'CreatePythonOperator', 'DEFAULT_REMAP_FUNCS', 'DataType', 'DeviceOption', 'DeviceScope', 'ExecutionStep', 'GetGlobalInitArgs', 'GetIndexFromGradientList', 'GlobalInit', 'GradGenMeta', 'GradientRegistry', 'GradientSlice', 'IR', 'InferBlobDevices', 'InferOpBlobDevices', 'InjectCrossDeviceCopies', 'InjectDeviceCopiesAmongNets', 'InjectDeviceCopiesAmongNetsWithoutB2D', 'IsOperator', 'IsOperatorWithEngine', 'NameScope', 'Net', 'OpSSA', 'OrderedDict', 'Plan', 'RefreshRegisteredOperators', 'RemapEntry', 'ScopedBlobReference', 'ScopedName', 'SetEnginePref', 'SetGlobalEnginePref', 'SetOpEnginePref', 'SetPerOpEnginePref', 'SparseGradGenMeta', '_GLOBAL_INIT_ARGS', '_GetRegisteredOperators', '_InitDataType', '_REGISTERED_OPERATORS', '_RectifyInputOutput', '_RegisterPythonImpl', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_add_net_to_dict', '_extract_stacktrace', '_get_blob_ref', '_recover_record_by_prefix', 'absolute_import', 'add_nets_in_order', 'basestring', 'binary_type', 'caffe2_pb2', 'chain', 'clone_and_bind_net', 'control_op_remap', 'copy_func_between_devices', 'defaultdict', 'device_equal', 'device_option_equal', 'division', 'execution_step', 'gen_do_gradient', 'gen_if_gradient', 'gen_while_gradient', 'get_net_name', 'get_op_ids_in_path', 'get_output_producers', 'get_ssa', 'get_undefined_blobs', 'namedtuple', 'np', 'output_to_list', 'pickle', 'print_function', 'recurrent_network_op_remap', 'remap_proto', 'scope', 'scoped_execution_step', 'string_types', 'sys', 'text_type', 'to_execution_step', 'unicode_literals', 'utils', 'viewitems', 'viewkeys', 'viewvalues', 'workspace']\n"
     ]
    }
   ],
   "source": [
    "# Importing core\n",
    "from caffe2.python import core\n",
    "printmd(\"**Core Imported**\")\n",
    "\n",
    "#List the various sub-functions inside the core operator\n",
    "x = list(dir(core))\n",
    "print (\"Parent Functions of core are: \\n\")\n",
    "print x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Creating operators\n",
    "\n",
    "Operators in caffe2 are mathematical functions on numerical arrays. These include every mathematical operation such as addition, subtraction etc; every booloean operation; all the layers generally involved in deep learning such as convolution, pooling , etc; and the data comparison functions. \n",
    "\n",
    "The function **core.CreateOperator** is used in instantiating every operator. Each of these operators, as we have seen in the previous video, as **protobuf objects**\n",
    "\n",
    "Below is a snippet to create an **Addition** operator, which takes in input as two tensors (numpy arrays) and applies elementwise addition operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type of the created op is: <class 'caffe2.proto.caffe2_pb2.OperatorDef'>\n",
      "Content:\n",
      "\n",
      "input: \"X1\"\n",
      "input: \"X2\"\n",
      "output: \"Y\"\n",
      "name: \"\"\n",
      "type: \"Add\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Creating the operator\n",
    "op = core.CreateOperator(\n",
    "    \"Add\",                                           # The type of operator that we want to run\n",
    "    [\"X1\", \"X2\"],                                    # Inputs\n",
    "    [\"Y\"],                                           # Output\n",
    ")\n",
    "\n",
    "print(\"Type of the created op is: {}\".format(type(op)))\n",
    "print(\"Content:\\n\")\n",
    "print(str(op))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: We will take each and every mathematatical operator, boolean operator, and data comparison operators in the **3rd video** of this section. The Layer operators will be dealt in the subsequent sections. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Initializing and managing networks\n",
    "\n",
    "Core has a sub-module, **Net**, which is used to model deep learning networks as computational graphs. Nets are essentially wrappers on base layers and operators. \n",
    "\n",
    "when we have a net, you can direct create an operator and add it to the net at the same time using Python tricks: essentially, if you call net.SomeOp where SomeOp is a registered type string of an operator, this essentially gets translated to\n",
    "    \n",
    "    op = core.CreateOperator(\"SomeOp\", ...)\n",
    "    net.Proto().op.extend([op])\n",
    "\n",
    "All the **Model Helper** functions are wrappers around this Net. We will take this up in the 4th video of this section where we will discuss Nets alogn with brew, model_helpers, cnn_helpers, etc.\n",
    "\n",
    "Below is a sample instantiation of Net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Naming the net - my_first_net\n",
      "\n",
      "\n",
      "Current network proto:\n",
      "name: \"my_first_net\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Creating a Net object\n",
    "net = core.Net(\"my_first_net\")\n",
    "print(\"Naming the net - my_first_net\\n\\n\")\n",
    "\n",
    "#Networks too are protobuf objects\n",
    "print(\"Current network proto:\\n{}\".format(net.Proto()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New network proto:\n",
      "\n",
      "name: \"my_first_net\"\n",
      "op {\n",
      "  output: \"X\"\n",
      "  name: \"\"\n",
      "  type: \"GaussianFill\"\n",
      "  arg {\n",
      "    name: \"std\"\n",
      "    f: 1.0\n",
      "  }\n",
      "  arg {\n",
      "    name: \"run_once\"\n",
      "    i: 0\n",
      "  }\n",
      "  arg {\n",
      "    name: \"shape\"\n",
      "    ints: 2\n",
      "    ints: 3\n",
      "  }\n",
      "  arg {\n",
      "    name: \"mean\"\n",
      "    f: 0.0\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Using a Gaussian Fill operator onto the network created above\n",
    "\n",
    "X = net.GaussianFill([], [\"X\"], mean=0.0, std=1.0, shape=[2, 3], run_once=0)\n",
    "print(\"New network proto:\\n\\n{}\".format(net.Proto()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Blob Rerencing and Namescope Assignments\n",
    "\n",
    "We will take this up later in this notebook after understanding workspaces and blobs. Refer the section: *Revisiting Blob Rerencing and Namescope Assignments*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Device Options\n",
    "\n",
    "Refer to the notebook introduced in the previous video."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "##################################################################################################\n",
    "##################################################################################################\n",
    "##################################################################################################\n",
    "##################################################################################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Workspace\n",
    "\n",
    "Similar to Matlab, workspaces store blobs, which in turn are numpy array objects stored in the memory. In general a blob is actually a typed pointer that can store any type of C++ objects, but Tensor is the most common type stored in a blob.\n",
    "\n",
    "Lets import workspace and see its sub-modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Workspace Imported**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ApplyTransform', 'ApplyTransformIfFaster', 'BenchmarkNet', 'Blobs', 'C', 'CallWithExceptionIntercept', 'CreateBlob', 'CreateNet', 'CurrentWorkspace', 'DeserializeBlob', 'FeedBlob', 'FeedImmediate', 'FetchBlob', 'FetchBlobs', 'FetchImmediate', 'GetCuDNNVersion', 'GetCudaPeerAccessPattern', 'GetDeviceProperties', 'GetNameScope', 'GetNetName', 'GetOperatorCost', 'GetStats', 'GlobalInit', 'HasBlob', 'ImmediateBlobs', 'InferShapesAndTypes', 'IsImmediate', 'Message', 'NumCudaDevices', 'Predictor', 'Process', 'RegisteredOperators', 'ResetWorkspace', 'RootFolder', 'RunNet', 'RunNetOnce', 'RunOperatorImmediate', 'RunOperatorOnce', 'RunOperatorsOnce', 'RunPlan', 'SerializeBlob', 'StartImmediate', 'StartMint', 'StopImmediate', 'StringifyBlobName', 'StringifyNetName', 'StringifyProto', 'SwitchWorkspace', 'WorkspaceGuard', 'Workspaces', '_BlobDict', '_Blob_feed', '_GetFreeFlaskPort', '_StringifyName', '_Workspace_create_net_with_exception_intercept', '_Workspace_run', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_immediate_mode', '_immediate_root_folder', '_immediate_workspace_name', 'absolute_import', 'basestring', 'blobs', 'caffe2_pb2', 'contextlib', 'defaultdict', 'division', 'has_gpu_support', 'is_asan', 'logger', 'logging', 'np', 'operator_tracebacks', 'os', 'print_function', 'scope', 'shutil', 'socket', 'tempfile', 'unicode_literals', 'utils']\n"
     ]
    }
   ],
   "source": [
    "# Importing workspace\n",
    "from caffe2.python import workspace\n",
    "printmd(\"**Workspace Imported**\")\n",
    "\n",
    "#List the various sub-functions inside the core operator\n",
    "x = list(dir(workspace))\n",
    "print x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Workspaces** enable loading of numpy data onto caffe2 blobs and manage them. Below are a few functions that will be frequently used during the entire course"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Workspace Sub-functions**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Workspace names**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reset the workspace\n",
      "Default workspace name: default\n",
      "Current workspace name: new_ws\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Loading and fetching blobs**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('List of all the blobs in workspace:', [u'X1', u'X2'])\n",
      "Input data    :  [  1.    3.   -5.    3.  -11.    4.5]\n",
      "Fetched data  :  [  1.    3.   -5.    3.  -11.    4.5]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "printmd(\"**Workspace Sub-functions**\")\n",
    "print(\"\\n\")\n",
    "\n",
    "\n",
    "######################################################################################################\n",
    "printmd(\"**Workspace names**\")\n",
    "\n",
    "#Reset all the workspaces (if created)\n",
    "print(\"Reset the workspace\")\n",
    "workspace.ResetWorkspace()\n",
    "\n",
    "#Switch workspace to Default one. \n",
    "workspace.SwitchWorkspace(\"default\")\n",
    "\n",
    "#Default workspace\n",
    "print(\"Default workspace name: {}\".format(workspace.CurrentWorkspace()))\n",
    "\n",
    "# Switch the workspace. The second argument \"True\" means creating the workspace if it is missing.\n",
    "# new_ws is the name of the new workspace\n",
    "workspace.SwitchWorkspace(\"new_ws\", True)\n",
    "print(\"Current workspace name: {}\".format(workspace.CurrentWorkspace()))\n",
    "\n",
    "#Switching back to default\n",
    "workspace.SwitchWorkspace(\"default\")\n",
    "print(\"\\n\")\n",
    "######################################################################################################\n",
    "\n",
    "\n",
    "######################################################################################################\n",
    "printmd(\"**Loading and fetching blobs**\") \n",
    "\n",
    "#Create numpy array \n",
    "input_data1 = [1.0, 3.0, -5.0, 3.0, -11.0, 4.5]\n",
    "input_data1 = np.asarray(input_data1)\n",
    "input_data1 = input_data1.astype(np.float32)  \n",
    "\n",
    "input_data2 = [True, False, False, True]\n",
    "input_data2 = np.asarray(input_data2)\n",
    "input_data2 = input_data2.astype(np.bool)\n",
    "\n",
    "#Load data into a workspace blob\n",
    "workspace.FeedBlob(\"X1\", input_data1)        #Blob has name X1\n",
    "workspace.FeedBlob(\"X2\", input_data2)        #Blob has name X2\n",
    "\n",
    "#List all the blobs in a workspace\n",
    "print(\"List of all the blobs in workspace:\", workspace.Blobs())\n",
    "\n",
    "#Fetch data from blob back to numpy array\n",
    "fetched_input_data1 = workspace.FetchBlob(\"X1\")\n",
    "print \"Input data    : \", input_data1\n",
    "print \"Fetched data  : \", fetched_input_data1\n",
    "\n",
    "#Two different workspaces can have blobs with same name and each of them can hold different data\n",
    "workspace.ResetWorkspace()\n",
    "workspace.FeedBlob(\"X1\", input_data1) # default workspace loads input_data1 onto blob X1\n",
    "workspace.SwitchWorkspace(\"new_ws\", True)\n",
    "workspace.FeedBlob(\"X1\", input_data2) # new_ws workspace loads input_data2 onto blob X1\n",
    "######################################################################################################\n",
    "\n",
    "workspace.SwitchWorkspace(\"default\")\n",
    "workspace.ResetWorkspace()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Revisiting - Blob Rerencing and Namescope Assignments\n",
    "\n",
    "**BlobReference** gives us a way to refer to the network that the blob is generated from. Note that blobs are, essentially, just strings in the current workspace. Ref: https://github.com/caffe2/caffe2/blob/master/caffe2/python/core.py\n",
    "\n",
    "Parametric manipulations and operator assignments over the blob can be done using the sub-functions of BlobReference. These are internally used by model helpers to construct network using the reference based chaining of operators  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "z1:  X1\n",
      "[u'APMeter', u'Abs', u'AbsGradient', u'Accumulate', u'AccumulateHistogram', u'Accuracy', u'Adagrad', u'Adam', u'Add', u'AddPadding', u'Alias', u'Allgather', u'Allreduce', u'And', u'Append', u'Assert', u'AtomicAppend', u'AtomicFetchAdd', u'AtomicIter', u'AveragePool', u'AveragePool1D', u'AveragePool1DGradient', u'AveragePool2D', u'AveragePool2DGradient', u'AveragePool3D', u'AveragePool3DGradient', u'AveragePoolGradient', u'AveragedLoss', u'AveragedLossGradient', u'BBoxTransform', u'BRGNCHWCToPackedInt8BGRAStylizerDeprocess', u'Barrier', u'BatchBoxCox', u'BatchBucketOneHot', u'BatchDenseToSparse', u'BatchGather', u'BatchGatherGradient', u'BatchMatMul', u'BatchOneHot', u'BatchSparseToDense', u'BatchToSpace', u'BooleanMask', u'BooleanMaskLengths', u'BooleanUnmask', u'BoxWithNMSLimit', u'Broadcast', u'Cast', u'Ceil', u'ChannelBackpropStats', u'ChannelShuffle', u'ChannelShuffleGradient', u'ChannelStats', u'CheckAtomicBool', u'CheckCounterDone', u'CheckDatasetConsistency', u'Checkpoint', u'Clip', u'ClipGradient', u'CloneCommonWorld', u'CloseBlobsQueue', u'CloseRebatchingQueue', u'Col2Im', u'CollectTensor', u'ColwiseMax', u'ColwiseMaxGradient', u'ComputeOffset', u'Concat', u'ConcatTensorVector', u'Conditional', u'ConditionalSetAtomicBool', u'ConstantFill', u'Conv', u'Conv1D', u'Conv1DGradient', u'Conv2D', u'Conv2DGradient', u'Conv3D', u'Conv3DGradient', u'ConvGradient', u'ConvTranspose', u'ConvTransposeGradient', u'Copy', u'CopyFromCPUInput', u'CopyOnDeviceLike', u'Cos', u'CosGradient', u'CosineEmbeddingCriterion', u'CosineEmbeddingCriterionGradient', u'CosineSimilarity', u'CosineSimilarityGradient', u'CountDown', u'CountUp', u'CpuUtilizationReport', u'CreateAtomicBool', u'CreateBlobsQueue', u'CreateBlobsQueueDB', u'CreateCommonWorld', u'CreateCounter', u'CreateDB', u'CreateMap', u'CreateMutex', u'CreateRebatchingQueue', u'CreateScope', u'CreateTensorVector', u'CreateTextFileReader', u'CreateTreeCursor', u'CrossEntropy', u'CrossEntropyGradient', u'DBExists', u'DepthConcat', u'DepthSplit', u'DequeueBlobs', u'DequeueRebatchingQueue', u'DestroyCommonWorld', u'DiagonalFill', u'Div', u'DivGradient', u'Do', u'DotProduct', u'DotProductGradient', u'DotProductWithPadding', u'DotProductWithPaddingGradient', u'Dropout', u'DropoutGrad', u'EQ', u'ElementwiseLinear', u'ElementwiseLinearGradient', u'Elu', u'EluGradient', u'EnqueueBlobs', u'EnqueueRebatchingQueue', u'EnsureCPUOutput', u'EnsureDense', u'Exp', u'ExpandDims', u'ExtendTensor', u'FC', u'FCGradient', u'FCTransposed', u'FCTransposedGradient', u'FeedBlob', u'FileStoreHandlerCreate', u'Find', u'FindDuplicateElements', u'Flatten', u'FlattenToVec', u'FlexibleTopK', u'FlexibleTopKGradient', u'FloatToFused8BitRowwiseQuantized', u'FloatToRowwiseQuantized8Bits', u'Floor', u'Free', u'Ftrl', u'Fused8BitRowwiseQuantizedToFloat', u'GE', u'GRUUnit', u'GRUUnitGradient', u'GT', u'Gather', u'GatherByKey', u'GatherPadding', u'GatherRanges', u'GatherRangesToDense', u'GaussianFill', u'GenerateProposals', u'GenerateProposalsCPP', u'GetAllBlobNames', 'GetNameScope', u'GivenTensorBoolFill', u'GivenTensorDoubleFill', u'GivenTensorFill', u'GivenTensorInt64Fill', u'GivenTensorIntFill', u'GivenTensorStringFill', u'Glu', u'HSoftmax', u'HSoftmaxGradient', u'HSoftmaxSearch', u'HasElements', u'HasScope', u'HuffmanTreeHierarchy', u'If', u'Im2Col', u'ImageInput', u'IndexFreeze', u'IndexGet', u'IndexHash', u'IndexLoad', u'IndexSize', u'IndexStore', u'InstanceNorm', u'InstanceNormGradient', u'IntIndexCreate', u'IsEmpty', u'IsMemberOf', u'Iter', u'KeySplit', u'KeyValueToMap', u'L1Distance', u'L1DistanceGradient', u'LC', u'LC1D', u'LC1DGradient', u'LC2D', u'LC2DGradient', u'LC3D', u'LC3DGradient', u'LCGradient', u'LE', u'LRN', u'LRNGradient', u'LSTMUnit', u'LSTMUnitGradient', u'LT', u'LabelCrossEntropy', u'LabelCrossEntropyGradient', u'LambdaRankNdcg', u'LambdaRankNdcgGradient', u'LastNWindowCollector', u'LayerNorm', u'LayerNormGradient', u'LeakyRelu', u'LeakyReluGradient', u'LearningRate', u'LengthsGather', u'LengthsIndicesInGradientSumGradient', u'LengthsMax', u'LengthsMaxWithMainInputAndForwardOutputGradient', u'LengthsMean', u'LengthsMeanGradient', u'LengthsPartition', u'LengthsRangeFill', u'LengthsSum', u'LengthsSumGradient', u'LengthsTile', u'LengthsToRanges', u'LengthsToSegmentIds', u'LengthsToShape', u'LengthsToWeights', u'LengthsTopK', u'LengthsTopKGradient', u'LengthsWeightedSum', u'LengthsWeightedSumGradient', u'LengthsWeightedSumWithMainInputGradient', u'Load', u'Log', u'Logit', u'LogitGradient', u'LongIndexCreate', u'LpNorm', u'LpNormGradient', u'LpPool', u'LpPoolGradient', u'MPIAllgather', u'MPIAllreduce', u'MPIBroadcast', u'MPICreateCommonWorld', u'MPIReceiveTensor', u'MPIReduce', u'MPISendTensor', u'MSRAFill', u'MakeTwoClass', u'MakeTwoClassGradient', u'MapToKeyValue', u'MarginRankingCriterion', u'MarginRankingCriterionGradient', u'MatMul', u'Max', u'MaxGradient', u'MaxPool', u'MaxPool1D', u'MaxPool1DGradient', u'MaxPool2D', u'MaxPool2DGradient', u'MaxPool3D', u'MaxPool3DGradient', u'MaxPoolGradient', u'MergeDim', u'MergeIdLists', u'Min', u'MinGradient', u'Mod', u'MomentumSGD', u'MomentumSGDUpdate', u'Mul', u'MultiClassAccuracy', u'NCHW2NHWC', u'NGramFromCategorical', u'NHWC2NCHW', u'NanCheck', u'NegateGradient', u'Negative', 'Net', u'Normalize', u'NormalizeGradient', u'NormalizeL1', u'NormalizePlanarYUV', u'Not', u'ONNXWhile', u'OneHot', u'Or', u'PRelu', u'PReluGradient', u'PackRNNSequence', u'PackRecords', u'PackSegments', u'PackedInt8BGRANHWCToNCHWCStylizerPreprocess', u'PadEmptySamples', u'PadImage', u'PadImageGradient', u'PairWiseLoss', u'PairWiseLossGradient', u'Partition', u'Percentile', u'Perplexity', u'PiecewiseLinearTransform', u'Pow', u'PrependDim', u'Print', u'Python', u'PythonDLPack', u'PythonDLPackGradient', u'PythonGradient', u'QuantDecode', u'QuantDecodeGradient', u'RMACRegions', u'Range', u'RangeFill', u'ReadNextBatch', u'ReadRandomBatch', u'ReceiveTensor', u'RecurrentNetwork', u'RecurrentNetworkBlobFetcher', u'RecurrentNetworkGradient', u'Reduce', u'ReduceBackMax', u'ReduceBackMaxGradient', u'ReduceBackMean', u'ReduceBackMeanGradient', u'ReduceBackSum', u'ReduceBackSumGradient', u'ReduceFrontMax', u'ReduceFrontMaxGradient', u'ReduceFrontMean', u'ReduceFrontMeanGradient', u'ReduceFrontSum', u'ReduceFrontSumGradient', u'ReduceFrontWeightedSum', u'ReduceFrontWeightedSumGradient', u'ReduceScatter', u'ReduceTailSum', u'Relu', u'ReluGradient', u'RemoveDataBlocks', u'RemovePadding', u'ReplaceNaN', u'ReservoirSampling', u'ResetCounter', u'ResetCursor', u'Reshape', u'ResizeLike', u'ResizeNearest', u'ResizeNearestGradient', u'RetrieveCount', u'ReversePackedSegs', u'RmsProp', u'RoIAlign', u'RoIAlignGradient', u'RoIPool', u'RoIPoolGradient', u'RowMul', u'RowWiseSparseAdagrad', u'RowWiseSparseAdam', u'Rowwise8BitQuantizedToFloat', u'RowwiseMax', u'RowwiseMaxGradient', u'SafeDequeueBlobs', u'SafeEnqueueBlobs', u'Save', u'Scale', u'ScatterAssign', u'ScatterWeightedSum', u'SegmentIdsToLengths', u'SegmentIdsToRanges', u'SegmentOneHot', u'Selu', u'SeluGradient', u'SendTensor', u'SequenceMask', u'Shape', u'Sigmoid', u'SigmoidCrossEntropyWithLogits', u'SigmoidCrossEntropyWithLogitsGradient', u'SigmoidGradient', u'Sign', u'Sin', u'SinGradient', u'SinusoidPositionEncoding', u'Size', u'Slice', u'SliceGradient', u'Snapshot', u'Softmax', u'SoftmaxGradient', u'SoftmaxWithLoss', u'SoftmaxWithLossGradient', u'Softplus', u'SoftplusGradient', u'Softsign', u'SoftsignGradient', u'SortAndShuffle', u'SortedSegmentMean', u'SortedSegmentMeanGradient', u'SortedSegmentRangeLogMeanExp', u'SortedSegmentRangeLogMeanExpGradient', u'SortedSegmentRangeLogSumExp', u'SortedSegmentRangeLogSumExpGradient', u'SortedSegmentRangeMax', u'SortedSegmentRangeMaxGradient', u'SortedSegmentRangeMean', u'SortedSegmentRangeMeanGradient', u'SortedSegmentRangeSum', u'SortedSegmentRangeSumGradient', u'SortedSegmentSum', u'SortedSegmentSumGradient', u'SortedSegmentWeightedSum', u'SortedSegmentWeightedSumGradient', u'SpaceToBatch', u'SparseAdagrad', u'SparseAdam', u'SparseFtrl', u'SparseLengthsIndicesInGradientSumGradient', u'SparseLengthsIndicesInGradientWeightedSumGradient', u'SparseLengthsIndicesInGradientWeightedSumWithMainInputGradient', u'SparseLengthsMean', u'SparseLengthsMean8BitsRowwise', u'SparseLengthsMeanFused8BitRowwise', u'SparseLengthsMeanGradient', u'SparseLengthsSum', u'SparseLengthsSum8BitsRowwise', u'SparseLengthsSumFused8BitRowwise', u'SparseLengthsSumGradient', u'SparseLengthsWeightedMean8BitsRowwise', u'SparseLengthsWeightedSum', u'SparseLengthsWeightedSum8BitsRowwise', u'SparseLengthsWeightedSumFused8BitRowwise', u'SparseLengthsWeightedSumGradient', u'SparseLengthsWeightedSumWithMainInputGradient', u'SparseMomentumSGDUpdate', u'SparseNormalize', u'SparseSortedSegmentMean', u'SparseSortedSegmentMeanGradient', u'SparseSortedSegmentSum', u'SparseSortedSegmentSumGradient', u'SparseSortedSegmentWeightedSum', u'SparseSortedSegmentWeightedSumGradient', u'SparseToDense', u'SparseToDenseMask', u'SparseToDenseMaskGradient', u'SparseUnsortedSegmentMean', u'SparseUnsortedSegmentMeanGradient', u'SparseUnsortedSegmentSum', u'SparseUnsortedSegmentSumGradient', u'SparseUnsortedSegmentWeightedSum', u'SparseUnsortedSegmentWeightedSumGradient', u'SpatialBN', u'SpatialBNGradient', u'SpatialSoftmaxWithLoss', u'SpatialSoftmaxWithLossGradient', u'Split', u'Sqr', u'SquareRootDivide', u'SquaredL2Distance', u'SquaredL2DistanceGradient', u'Squeeze', u'StatRegistryCreate', u'StatRegistryExport', u'StatRegistryUpdate', u'StopGradient', u'StoreAdd', u'StoreGet', u'StoreSet', u'StoreWait', u'StringEndsWith', u'StringIndexCreate', u'StringJoin', u'StringPrefix', u'StringStartsWith', u'StringSuffix', u'Sub', u'Sum', u'SumElements', u'SumElementsGradient', u'SumInt', u'SumReduceLike', u'SumSqrElements', u'Summarize', u'Swish', u'SwishGradient', u'TT', u'TTLinearGradient', u'Tanh', u'TanhGradient', u'TensorProtosDBInput', u'TensorVectorSize', u'TextFileReaderRead', u'ThresholdedRelu', u'ThresholdedReluGradient', u'Tile', u'TileGradient', u'TimerBegin', u'TimerEnd', u'TimerGet', u'TimerGetAndEnd', u'TopK', u'TopKGradient', u'Transpose', u'TrimDataset', u'UnPackRecords', u'UniformFill', u'UniformIntFill', u'Unique', u'UniqueUniformFill', u'UnpackRNNSequence', u'UnpackSegments', u'UnsafeCoalesce', u'UnsortedSegmentMean', u'UnsortedSegmentMeanGradient', u'UnsortedSegmentSum', u'UnsortedSegmentSumGradient', u'UnsortedSegmentWeightedSum', u'UnsortedSegmentWeightedSumGradient', u'WallClockTime', u'WeightedSample', u'WeightedSampleDequeueBlobs', u'WeightedSigmoidCrossEntropyWithLogits', u'WeightedSigmoidCrossEntropyWithLogitsGradient', u'WeightedSum', u'WeightedSumGradient', u'Where', u'While', u'XavierFill', u'Xor', u'YellowFin', u'ZeroGradient', '_CreateAndAddToNet', '__add__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__getattr__', '__getattribute__', '__hash__', '__init__', '__module__', '__ne__', '__new__', '__radd__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_from_net', '_name', 'meta', u'rnn_internal_accumulate_gradient_input', u'rnn_internal_apply_link']\n"
     ]
    }
   ],
   "source": [
    "input_data1 = [1.0, 3.0, -5.0, 3.0, -11.0, 4.5]\n",
    "input_data1 = np.asarray(input_data1)\n",
    "input_data1 = input_data1.astype(np.float32) \n",
    "\n",
    "workspace.FeedBlob(\"X1\", input_data1)\n",
    "\n",
    "z1 = core.BlobReference(\"X1\")\n",
    "print \"z1: \", z1\n",
    "print dir(z1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those who are familiar with tensorflow would be aware of **NameScopes** (Variable Scopes). Name scopes help in data management in two ways\n",
    "\n",
    "1) DeviceScopes - Defining which computing platform holds what data.\n",
    "\n",
    "2) NameScopes - Helping management of data storage with unique identifier names.\n",
    "\n",
    "A sample example of namescope is below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Understanding NameScope**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current Blobs in the workspace:  [u'ns1/X1', u'ns1/X2', u'ns1/Y', u'ns2/X1', u'ns2/X2', u'ns2/Y']\n"
     ]
    }
   ],
   "source": [
    "printmd(\"**Understanding NameScope**\")\n",
    "\n",
    "#Creating a sinusoidal input data\n",
    "Fs = 8000\n",
    "f = 5\n",
    "sample = 8000\n",
    "x = np.arange(sample)\n",
    "input_data1 = np.sin(2 * np.pi * f * x / Fs)\n",
    "input_data2 = np.cos(2 * np.pi * f * x / Fs)\n",
    "\n",
    "workspace.ResetWorkspace()\n",
    "with core.NameScope(\"ns1\"):\n",
    "    op = core.CreateOperator(\n",
    "        \"Add\",                                           # The type of operator that we want to run\n",
    "        [\"X1\", \"X2\"],                                           # Inputs\n",
    "        [\"Y\"],                                           # Output\n",
    "    )\n",
    "\n",
    "workspace.FeedBlob(\"ns1/X1\", input_data1)\n",
    "workspace.FeedBlob(\"ns1/X2\", input_data2)\n",
    "\n",
    "workspace.RunOperatorOnce(op)                       #Running the operator once\n",
    "\n",
    "\n",
    "with core.NameScope(\"ns2\"):\n",
    "    op = core.CreateOperator(\n",
    "        \"Add\",                                           # The type of operator that we want to run\n",
    "        [\"X1\", \"X2\"],                                           # Inputs\n",
    "        [\"Y\"],                                           # Output\n",
    "    )\n",
    "\n",
    "workspace.FeedBlob(\"ns2/X1\", input_data1)\n",
    "workspace.FeedBlob(\"ns2/X2\", input_data2)\n",
    "\n",
    "workspace.RunOperatorOnce(op)                       #Running the operator once\n",
    "\n",
    "print \"Current Blobs in the workspace: \", workspace.Blobs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Brew - Model_Helpers - CNN_Helpers - Net Operations - Model Downloaders - Predictor - Examples\n",
    "\n",
    "\n",
    "We will discuss on the basics of these functions and modules in the **4th** video of this section. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 Base Layers - Base Optimizers - Base Regularizers\n",
    "\n",
    "These modules will be taken up inidividually along with their subsequent theories in the later sections of this course. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Memory Manager\n",
    "\n",
    "This sub module provides access to (Helper functions access these internal functions to provide summaries)\n",
    "\n",
    "1) Estimating memory usage of model. \n",
    "\n",
    "2) Releasing non-required data blobs' memory during execution.\n",
    "\n",
    "3) Finding the source and target node locations during data transfer\n",
    "\n",
    "4) Enabling weight sharing between layers\n",
    "\n",
    "5) Assigning best possible paths for data transfer amongst the blobs and keeping a track of it. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Distributed Computation\n",
    "\n",
    "This will be dealt in details during the last section of this course. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
